Method and apparatus for generating quality estimators

ABSTRACT

A system that incorporates teachings of the present disclosure may include, for example, sampling a variable effect distribution of viewing preference data to determine a first set of effects comprising a plurality of first distortion type effects associated with a first distortion type of a first image and to determine a second set of effects comprising a plurality of second distortion type effects associated with the second distortion type of a second image, calculating a preference estimate from a logistic regression model of the viewing preference data according to the first set of effects and the second set of effects, wherein the preference estimate comprises a probability that the first image is preferred over the second image, and selecting one of the first distortion type or the second distortion type according to the preference estimate. Other embodiments are disclosed.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to image and video quality andmore specifically to generating image and video quality estimators.

BACKGROUND

Quality control for images and video is often implemented based onsubjective testing using people that compare different images or videocontent. A quality estimator can be used to compute a score to estimateperceived quality of a single input image. When comparing image qualityestimate scores of two images, there can be is uncertainty as to how tocompare such scores or what fraction of viewers might actually preferthe image with the lower score.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustrative embodiment of a system that providescommunication services;

FIGS. 2A and 2B depict an illustrative embodiment of an image pair thatis used for performing subjective evaluation;

FIGS. 3A and 3B depict an illustrative embodiment of interval estimatesof distortion effects and total quality estimator effects of viewingpreference data;

FIGS. 4A and 4B depicts an illustrative embodiment of binned residualplots demonstrating goodness-of-fit of a logistic regression model;

FIGS. 5, 6, 7A and 7B depict an illustrative embodiments of a methodoperating in portions of the system described in FIG. 1

FIG. 8 depicts an illustrative embodiment of density estimates forposterior distribution of probabilities of choosing images from imagepairs; and

FIG. 9 is a diagrammatic representation of a machine in the form of acomputer system within which a set of instructions, when executed, maycause the machine to perform any one or more of the methods discussedherein.

DETAILED DESCRIPTION

The present disclosure describes, among other things, systems,components and/or methods for predicting image preferences according toquality estimator (QE) modeling. In one or more embodiments, apredictive model can estimate a probability that one image of an imagepair will be preferred by a random viewer relative to a second image ofthe pair. The model can be trained using results from a large-scalesubjective test and can present the degree to which various factorsinfluence subjective quality. In one or more embodiments, the model canprovide well-calibrated estimates of pairwise image preferences using avalidation set of image pairs.

In one or more embodiments, a logistic regression model can be generatedfrom viewing preference data. The viewing preference data can begenerated by preparing reference images. The reference images can beprocessed according to distortion types to create distorted images. Inone or more embodiments, distorted images can be presented to viewers,in pairs, and the viewers who can pick which image is of better quality.The data from the viewing preferences can be modeled via a logisticregression model.

In one or more embodiments, the logistic regression model can be usedpredict which image of a pair of images will likely be preferred overthe other. The model can take into account image effects, viewereffects, distortion type effects, and quality estimator effects tocalculate a predictive probability that an image that has been generatedvia a particular distortion type will be preferred over another image ina viewing pair. In one or more embodiments, the model can calculate aconfidence level for the probability. In one or more embodiments, thepredictive probability can be used to select distortion types, whichwill correspond to anticipated viewer preferences.

One embodiment of the present disclosure includes a server having amemory and a processor. The memory has executable instructions and theprocessor circuit is coupled with the memory. The processor, whenexecuting the executable instructions, can perform operations forreceiving a request over a network to predict an image preference for animage pair comprising a first image and a second image. The processorcan also perform operations for identifying a first distortion typeassociated with the first image and a second distortion type associatedwith the second image. The processor can, in turn, perform operationsfor sampling a variable effect distribution of viewing preference datato determine a first set of effects and a second set of effects. Thefirst set of effects can include a plurality of first reference imageeffects associated with the first image and a plurality of firstdistortion type effects associated with the first distortion type. Thesecond set of effects can include a plurality of second reference imageeffects associated with the second image and a plurality of seconddistortion type effects associated with the second distortion type. Theprocessor can perform operations for calculating a preference estimatefrom a logistic regression model of the viewing preference dataaccording to the first set of effects and the second set of effects. Thepreference estimate can include a probability that the first image ispreferred over the second image. In turn, the processor can performoperations for sending the preference estimate over the network.

One embodiment of the present disclosure is a method includingidentifying, by a processor including a device, a first distortion typeassociated with a first image of an image pair and a second distortiontype associated with a second image of the image pair. The method canfurther include sampling, by the processor, a variable effectdistribution of viewing preference data to determine a first set ofeffects comprising a plurality of first distortion type effectsassociated with the first distortion type. The sampling can also includedetermining a second set of effects comprising a plurality of seconddistortion type effects associated with the second distortion type. Themethod can include calculating, by the processor, a preference estimatefrom a logistic regression model of the viewing preference dataaccording to the first set of effects and the second set of effects. Thepreference estimate can include a probability that the first image ispreferred over the second image. The method can, in turn, includeselecting, by the processor, a distortion type for processing mediacontent from a plurality of distortion types according to the preferenceestimate.

One embodiment of the present disclosure is a computer-readable storagemedium including executable instructions, which when executed by aprocessor cause the processor to perform operations including sampling avariable effect distribution of viewing preference data to determine afirst set of effects including a plurality of first distortion typeeffects associated with a first distortion type of a first image. Thesampling can further determine a second set of effects comprising aplurality of second distortion type effects associated with the seconddistortion type of a second image. The instructions can further causethe processor to perform operations for calculating a preferenceestimate from a logistic regression model of the viewing preference dataaccording to the first set of effects and the second set of effects. Thepreference estimate can include a probability that the first image ispreferred over the second image. The instructions can, in turn, causethe processor to perform operations for selecting one of the firstdistortion type or the second distortion type according to thepreference estimate.

FIG. 1 depicts an illustrative embodiment of a system 100 for providingpredictions of viewer preferences between two images in an image pair.The system 100 can access a model, based on viewing preference data. Thesystem 100 can used the model to determine a probability that a firstimage will be preferred over a second image and a confidence level forthis prediction. The system 100 can be used to select between competingimages and/or video. The system 100 can be used to select betweencompeting devices and/or methods for processing, transmitting, andreproducing images and video by predicting which of the competingdevices and/or methods will be preferred by viewers.

The system 100 can represent an Internet Protocol Television (IPTV)media system. The IPTV media system can include a super head-end office(SHO) 110 with at least one super headend office server (SHS) 111 whichreceives content from satellite and/or terrestrial communicationsystems. In the present context, content can represent in whole or inpart, for example, messages, text, audio, moving images such as 2D or 3Dvideos, video games, virtual reality content, still image content, andcombinations thereof. The SHS server 111 can forward packets associatedwith the media content to one or more video head-end servers (VHS) 114via a network of video head-end offices (VHO) 112 according to a commonmulticast communication protocol.

The VHS 114 can distribute multimedia content, including broadcastcontent, via an access network 118 to commercial and/or residentialbuildings 102 housing a gateway 104 (such as a residential or commercialgateway). The access network 118 can represent a group of digitalsubscriber line access multiplexers (DSLAMs) located in a central officeor a service area interface that provide broadband services over fiberoptical links or copper twisted pairs 119 to buildings 102. The gateway104 can use common communication technology to distribute broadcastsignals to media processors 106 such as Set-Top Boxes (STBs) which inturn present broadcast channels to media devices 108 such as computersor television sets managed in some instances by a media controller 107(such as an infrared or RF remote control). Other data can bedistributed to the media processors 106 via the gateway, including voicemessages, text messages, voice communications, video conferencing andcombinations thereof.

The gateway 104, the media processors 106, and/or media devices 108 canutilize tethered communication technologies (such as coaxial, powerlineor phone line wiring) or can operate over a wireless access protocolsuch as Wireless Fidelity (WiFi), Bluetooth, Zigbee, or other present ornext generation local or personal area wireless network technologies. Byway of these interfaces, unicast communications can also be invokedbetween the media processors 106 and subsystems of the IPTV media systemfor services such as video-on-demand (VoD), browsing an electronicprogramming guide (EPG), or other infrastructure services.

A satellite broadcast television system 129 can also be used in thesystem of FIG. 1. The satellite broadcast television system can beoverlaid, operably coupled with, or replace the IPTV system as anotherrepresentative embodiment of communication system 100. In thisembodiment, signals transmitted by a satellite 115 carrying mediacontent can be received by a satellite dish receiver 131 coupled to thebuilding 102. Modulated signals received by the satellite dish receiver131 can be transferred to the media processors 106 for demodulating,decoding, encoding, and/or distributing broadcast channels to the mediadevices 108. The media processors 106 can be equipped with a broadbandport to the ISP network 132 to enable interactive services such as VoDand EPG as described above.

In yet another embodiment, an analog or digital cable broadcastdistribution system such as cable TV system 133 can be overlaid,operably coupled with, or replace the IPTV system and/or the satelliteTV system as another representative embodiment of communication system100. In this embodiment, the cable TV system 133 can also provideInternet, telephony, and interactive media services.

The embodiments of the present disclosure can apply to otherover-the-air and/or landline media content services system.

Some of the network elements of the IPTV media system can be coupled toone or more computing devices 130, a portion of which can operate as aweb server for providing web portal services over an Internet ServiceProvider (ISP) network 132 to wireline and/or wireless devices,including media devices 108 and/or portable communication devices 116.

Multiple forms of media services can be offered to media devices overlandline technologies in communication system 100 such as through thedevices and/or techniques described above. Additionally, media servicescan be offered to media devices by way of a wireless access base station117 operating according to common wireless access protocols such asGlobal System for Mobile or GSM, Code Division Multiple Access or CDMA,Time Division Multiple Access or TDMA, Universal MobileTelecommunications or UMTS, World interoperability for Microwave orWiMAX, Software Defined Radio or SDR, Long Term Evolution or LTE, and soon. Other present and next generation wide area wireless networktechnologies are contemplated by the present disclosure.

System 100 can also provide for all or a portion of the computingdevices 130 to function as a probability prediction server for relativequality estimation (herein referred to as server 130). The server 130can use common computing and communication technology to performfunction 162, which can include among things, receiving a request topredict an image preference between two images, identifying competingimages, viewers, and/or distortion types that are being compared,sampling a variable effect distribution based on viewing preference datato determine sets of effects for the competing images, viewers, and/ordistortion types, and calculating a probability that one image will bepreferred over the other image based on the sets of effects. Theexemplary embodiments can utilize any number of servers 130 which canimplement any number of evaluators 162 for the QEs.

The system 100 can perform image and video capture, compression,transmission, and display. In one embodiment, the server 130 canaccurately predict subjective, human quality judgments for a wide rangeof input content and processing types. The server 130 can accuratelymimic human responses in a probabilistic manner using apairwise-preference predictor which can provide an estimate of the form“a random viewer will prefer the image on the left with a probabilityof, e.g., 40%”. In one embodiment, the server 130 can also attach amargin of error to the estimate. For example, for a group of 100viewers, the server 130 can provide an interval prediction (i.e. 40+/−6)of the number of viewers who will prefer the left image to the right.The pair-wise probability estimate can out-perform an absolute qualityestimate, which can expensive and time consuming to generate and can belimited by human responses that are inherently probabilistic.Inter-viewer variability, based on individual viewer abilities todiscriminate, preferences between distortion types, and/or distortionsin particular spatial regions of images, can cause unreliable resultsfor an absolute score quality estimate.

In one or more embodiments, the server 130 can model relative quality,in the form of a relative prediction that “an image on the LEFT isbetter than that one on the RIGHT.” The server 130 can provide arelative quality estimate prediction, as opposed to an absolute qualityscore (e.g., “this image has a quality score of 4.5”). In one or moreembodiments, the relative quality estimate can be provided for productand algorithm benchmarking and selection as described, for example, in“No-reference image and video quality estimation: Applications andhuman-motivated design,” by S. S. Hemami and A. R. Reibman, in SignalProcessing: Image Communication, August 2010, the disclosure of which ishereby incorporated by reference.

In one or more embodiments, the server 130 provides a relative qualityestimate that allows for a probabilistic interpretation. A probabilisticinterpretation provides more context than an absolute quality estimate.For example, knowing that two images have absolute quality estimatescores of 2.3 vs. 3.6 is less informative, taken alone, than knowingthat 75% of viewers prefer the first image over the second image. Thisrelative estimate of quality can be interpreted by a layperson, whilethe absolute quality estimate only has meaning to an expert. In one ormore embodiments, the predictive model used by server 130 can be basedon large scale subjective testing. Viewer preference data can becollected for the generating the predictive model. This data collectioncan be based on posing a straightforward question of relative preferencethat can be easily and competently completed by lightly trained viewersas more fully described in “A crowd sourced QoE evaluation framework formultimedia content,” by K. T. Chen, C. C. Wu, Y. C. Chang, and C. L.Lei, in Proceedings of the 17th ACM international conference onMultimedia. ACM, 2009, pp. 491-500, the disclosure of which is herebyincorporated by reference.

By comparison, subjective testing methods for absolute ratings requiretraining for viewers to recognize the meaning of a score and tocorrectly use a dynamic scoring range. This further implies thatabsolute score data collection is limited and the results thus obtainedmay not apply to real-life viewers and viewing conditions as well asresults obtained via relative preference testing. An absolute qualityestimate can be used to decide whether one image has a better (or worse)quality than another, where if their objective quality scores differ bymore than a constant threshold (Δo). However, this is contingent uponthe choice of an effective and meaningful Δo, which does not reflect theprobabilistic nature of viewer responses.

In one or more embodiments, the server 130 can provide a probabilisticpairwise-preference predictor (P4), which uses a statistical model forpairwise preferences that is a function of properties of degraded imagesincluding reference image, distortion type, and an ensemble of QEscores. In several embodiments, full-reference quality estimators can beused with the pair-wise probability predictor. For example, predictivemodel can be used with the quality estimator, SSIM, which is describedmore fully in “Image quality assessment: from error visibility tostructural similarity,” by Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P.Simoncelli, in IEEE Trans. Image Proc., vol. 13, no. 4, pp. 600-612,April 2004, and incorporated herein by reference in its entirety. Otherfull-reference quality estimators that can be used include, qualityestimator IW-SSIM, described in “Information content weighting forperceptual image quality assessment,” by Z. Wang and Q. Li, in IEEETrans. Image Processing, vol. 0, no. 5, pp. 1185-1198, May 2011,incorporated herein by reference in its entirety; quality estimatorPSNR-HVS-M, described in “On between-coefficient contrast masking of DCTbasis functions,” by N. Ponomarenko et al., in Workshop on VideoProcessing and Quality Metrics, 2007, incorporated herein by referencein its entirety; quality estimator VIF, described in “Image informationand visual quality,” by H. R. Sheikh and A. C. Bovik, in IEEE Trans.Image Proc., vol. 15, no. 2, pp. 430-444, February 2006, incorporatedherein by reference in its entirety; quality estimator VSNR, describedin “VSNR: A wavelet-based visual signal-to-noise ratio for naturalimages,” by D. M. Chandler and S. S. Hemami, in IEEE Trans. Image Proc.,vol. 16, no. 9, pp. 2284-2298, September 2007, incorporated herein byreference in its entirety; and quality estimator PSNR, described in “Howwell do line drawings depict shape?,” by F. Cole, K. Sanik, D. DeCarlo,A. Finkelstein, T. Funkhouser, S. Rusinkiewicz, and M. Singh, in ACMTrans. Graphics, vol. 28, no. 3, pp. 28:1-28:9, July 2009, andincorporated herein by reference in its entirety.

In one or more embodiments, the predictive model of the server 130 canbe based on subjective data that is collected from viewers of imagepairs. For example, the subjective data can be collected using an AmazonMechanical Turk (AMT), which is suited for the reduced trainingrequirements of relative preference data collection and the need forlarge-scale data collection. In one or more embodiments, the subjectivedata from image pairs can be collected using an image quality database.For example, the subjective viewing preferences may be collected usingany of the large, publicly available subjective image quality datasets,such as the LIVE, TID2008, and CSIQ databases, which are described in “Astatistical evaluation of recent full reference image quality assessmentalgorithms,” by H. R. Sheikh, M. F. Sabir, and A. C. Bovik, IEEE Trans.Image Proc., vol. 15, no. 11, pp. 3440-3451, November 2006, in“TID2008—a database for evaluation of full-reference visual qualityassessment metrics,” by N. Ponomarenko et al., in Advances of ModernRadioelectronics, vol. 10, pp. 30-45, 2009, and in “Most apparentdistortion: Full-reference image quality assessment and the role ofstrategy,” by E. C. Larson and D. M. Chandler, in the Journal ofElectronic Imaging, vol. 19, no. 1, March 2010, each referenceincorporated herein in its entirety.

In one or more embodiments, a collection of references, or sourceimages, can be chosen from a database of references for using collectingpair-wise preference data. Alternatively, the reference images can becaptured using a high quality, high-resolution digital camera. Inanother embodiment, each of the reference images can initially processedby, for example, filtering, down-sampling, and/or cropping, to produceimages of consistent size and resolution. The reference images can be ofsame or similar subject matter. Alternatively, the reference images caninclude a range of different subjects or scenes. For example, a set ofreference images can include animal pictures, landscapes, and structures(including buildings and sculptures). In one or more embodiments,spatial information (SI) and colorfulness (CF) scores can be capturedand computed for each reference image for inclusion into a database ofviewing preference data accessible to the server 130.

In one or more embodiments, distortion types can be selected forapplication to the reference images that are subjectively assessed. Forexample, the reference images can be processed using such as Gaussianblur, JPEG-2000 and JPEG compression, and additive Gaussian noise. Foreach distortion type, a range of severity values, from little distortionto moderately severe, can also be selected. The multiplicity ofdistortion types and distortion severities can result in a large numberof distorted images for each reference image and, by extension, a largenumber of total distorted images for the overall image set that isavailable for the subjective testing. In an exemplary collection set, atotal of 119 distorted images can be generated for each of 90 referenceimages by using a multiplicity of distortion types and distortionseverities. For example, FIGS. 2A and 2B depict an illustrativeembodiment of an image pair that is used for performing subjectivetesting. FIG. 2A depicts a first version of a reference image, with afirst distortion, while FIG. 2B depicts a second version of the samereference image with a second distortion. While participating in thesubjective test, a viewer declares a preference between the left andright versions of the reference image.

In one embodiment, the viewing preference data is collected by settingup a two-stage subjective test. In the first stage, a first set ofdistorted reference images can be used to generate data for training thepredictive model. In the second stage, a second set of distortedreference images can be used for validating the model. For example, afirst set of 3550 distorted images can be generated from a CSIQ imagedatabase of 30 images by applying differing types and degrees ofdistortion to the images. A first set of labeled pairs can be generatedfrom the 3550 distorted images and used for subjective preferencetesting by viewers. The results of the subjective preference testing arethen used for training a logistic regression model for predictingpreferences. Next, a second set of 10,690 distorted images can begenerated from 90 new images that are subjected to a set of availabledistortions. This second set of distorted images can be used to generatea second set of labeled pairs. The second set of labeled pairs can thenbe used to refine validate the logistic regression model that has beengenerated.

In one or more embodiments, the labeled pairs of images for thesubjective testing can be chosen to emphasize important use cases of aQE. In one embodiment, all possible pairs can be decomposed into foursets, based on whether both images share a common reference image or acommon distortion type. The subjective testing can contains comparisonsacross different reference images. The image pairs can be presented totest volunteers (viewers) in random order and/or random left/rightassignment. In one embodiment, test subjects can be instructed to,simply, “click on the image with better visual QUALITY between the twoimages. Choose the image with the better technical quality, not theimage content you prefer.” In one embodiment, the subjective test caninclude sets of pairs, where each viewer can limited to a maximum numberof pairs. In an exemplary embodiment, a group of 450 unique viewersparticipated in a subjective study, with no image pair being rated bymore than one viewer. Data from viewers whose data was clearlyunreliable or showed extreme bias was rejected.

In one or more embodiments, a logistic regression model can be used topredict which image of an image pair will be chosen by a viewer. Forexample, viewer preference data that has been generated from large-scalesubjective testing can be used to fit and/or train a multilevelBradley-Terry model. In one or more embodiments, a latent subjectivequality of each image can be modeled as a function of image levelvariables, such as the reference image, the distortion type applied tothe image, and/or one or more quality estimators, where the effect foreach quality estimator differs by distortion type. In one or moreembodiments, modeling the subjective quality as a function ofimage-level variables allows the model to be applied to images outsideof those used for training data. In one or more embodiments, the trainedmultilevel logistic regression model can be used by the server 130 tomake predictions about viewer preferences for new image pairs. In oneembodiment, the server 130 fits and/or trains the large-scale subjectivetest data to the multilevel logistic regression model. In anotherembodiment, the server 130 can receive the model after it has beenfitted and/or trained by a different part of the system 100.

In one embodiment, the model can be configured with a result variable,Y. For an image pair i comprising a right image and a left image,Y_(i)=1, if the subject chose the left image, and Y_(i)=0, if thesubject chose the right image, for image pairs i=1, . . . , N. In anexemplary embodiment, a total of N=13,674 image pairs were used from thesubjective testing, representing 80% of the image pairs from stage 1 ofthe experiment, while the remaining 20% were held out for validationtesting. In one embodiment, viewer variable, V[i], denotes the viewer ofimage pair i, for viewers w=1, etc. In the exemplary embodiment, a totalof 249 viewers participated in the subjective testing. In anotherembodiment, reference images for the left and right images are denotedas L[i] and R[i], respectively, in pair i, for reference images j=1, . .. , J. In the exemplary embodiment, a total of 30 reference images wereused in the subjective testing. In one embodiment, distribution typevariables, Dist-L[i] and Dist-R[i], denote the distortion types appliedto the left and right images in pair i, for distortion types d=1, . . ., D. In the exemplary embodiment, a total of 4 distortion types wereused. In another embodiment, the variables X_(k[i]) ^(QE-L) and X_(k[i])^(QE-R) can be objective quality scores for the kth QE applied to theleft and right images in pair i, respectively, for QEs k=1, . . . , K=6.In the exemplary embodiment, six different quality estimator scores wereincluded. In one embodiment, all of the QE scores can be transformed andscaled to have a mean of zero and a standard deviation of one such thatthe estimated effects for each of the QE scores are comparable, whereeach increase monotonically with image quality.

In one or more embodiments, a model can be defined for predictingprobability that a viewer will select a particular image of the imagepair. In one example, a multilevel (i.e. hierarchical) Bayesian logisticregression model can be used (Equation 1):

${Y_{i} \sim {{Bemoulli}\left( p_{i} \right)}},{{\log\left( \frac{p_{i}}{1 - p_{i}} \right)} = {{\alpha_{V{\lbrack i\rbrack}}^{viewer} + \lambda_{i}^{Left} - {\lambda_{i}^{Right}\lambda_{i}^{Left}}} = {\alpha_{D{\lbrack i\rbrack}}^{image} + \alpha_{{DISt} - {L{\lbrack i\rbrack}}}^{{distortion} - {type}} + {\sum\limits_{k = 1}^{K}\;{\beta_{{({k,{{Dist} - L}})}{\lbrack i\rbrack}}^{objective} \times X_{k{\lbrack i\rbrack}}^{{QE} - L}}}}}},$for image pairs i=1, . . . , N, where λ_(i) ^(Right) is definedanalogously to λ_(i) ^(Left) and where λ_(i) ^(Right) and λ_(i) ^(Left)represent latent subjective qualities of the right and left images,respectively. Note that λ_(i) ^(Right) and λ_(i) ^(Left) are functionsof lower level characteristics of the images, such as their referenceimages, distortion types, and QE scores.

In one or more embodiments, normal priors can be used for the viewereffects, the reference image effects, the distortion type effects, andQE effects. For example, the priors can be:α_(v) ^(viewer) ˜N(μ^(viewer),σ_(viewer) ²),α_(r) ^(image) ˜N(0,σ_(image) ²),α_(d) ^(distortion-type) ˜N(0,σ_(distortion-type)),β_(hd) ^(objective) ˜N(μ_(k) ^(objective),τ_(objective-dist) _(h) )μ_(k) ^(objective) ˜N(μ₀,τ_(objective) ²),In another embodiment, weakly informative half-t priors can be used forthe standard deviation parameters σ_(viewer), σ_(image),σ_(distortion-type), and τ_(objective-dist k) (for k=1, . . . , 6), andτ_(objective), while N(0, 1) priors can be used for μ_(viewer) and μ₀,as further described in “A weakly informative default prior distributionfor logistic and other regression models,” by A. Gelman, A. Jakulin, M.G. Pittau, and Y. S. Su, in Annals of Applied Statistics, vol. 2, no. 4,pp. 1360-1383, 2008, incorporated by reference herein in its entirety.

In the exemplary embodiment, inferences can be made for point estimatesand interval estimates for various parameters in the model. For example,the mean left/right bias (μ_(viewer)) in the population of subjects forthe subjective testing was about 0.06 (on a logistic scale). Thiscorresponds to a probability of a random viewer picking the left imagewith probability 51.5%. The viewer bias effect, however, was smallcompared to the effects of the other factors in the model. The estimatedstandard deviations for the viewer, reference image, and distortion typeeffects were σ_(viewer)=0.19(0.04), σ_(image)=0.44 (0.07), andα_(distortion)=1.03 (0.60), respectively, where the standard errors arelisted in parentheses. As can be seen, of these three factors, thedistortion type explained the most variation in the outcome, while theviewer bias explained the least variation.

To interpret these group level standard deviations, consider that on thelogistic scale, holding all other variables at their observed values,choosing two different distortion types at random would induce anexpected change in the probability of choosing the left image of about25%—a large effect. However, randomly choosing two reference images, ortwo viewers, would affect the probability of choosing the left image byabout 12% or 5%, on average, respectively.

In the exemplary embodiment, for the most extreme variations, the mostbiased viewers in our experiment had approximately a 44% and 59%probability of choosing the left image, holding all other variablesconstant. The most preferred reference image, all else held constant,was “sunset color” (74.0% chance of being preferred compared to theaverage reference image), and the least preferred was “fisher” (26.7%).The estimated effects for distortion types and QEs in the exemplaryembodiment are pictured in FIGS. 3A and 3B. FIG. 3A shows that JPEG andJPEG2000 distortions were preferred over Blur and Noise distortionsacross the images in the subjective testing, where a JPEG-distortedimage would be preferred over a noise-distorted image (all else heldconstant) about 80% of the time, for example. In the exemplaryembodiment, 24 objective quality effects were generated and denoted bythe parameter set, β_(kd) ^(objective) (for k=1, . . . , 6 and d=1, . .. , 4). The objective quality effects were more complicated tointerpret, since all six QEs are highly correlated with each other. Tosummarize each of the effects, a posterior distribution of the sum oftheir effects can be estimated for each distortion type. These estimatesof these sums demonstrate values of 1.50, 1.86, 0.57, and 1.86 for thedistortion types Blur, JPEG2000, JPEG, and Noise, respectively, withstandard errors less than 0.12 in all four cases, as shown in FIG. 3B.In the exemplary data, the six QEs demonstrated the strongestassociation with the subjective quality for the JPEG2000 and Noisedistortion types, and the weakest association for the JPEG and blurdistortions.

In a further embodiment, the fit of the model can be verified by makingpredictions on a holdout set, which can consist of a percentage of thepairs in the data from Stage 1. For each image pair in the holdout set,and for each posterior sample of parameters, an estimated can becomputed of the probability that the viewer chose the left image. Aresponse can be simulated from the Bernoulli distribution according tothis probability. From these simulations, binned residual plots can bedraw to examine the fit of the model.

FIG. 4A depicts an illustrative embodiment of the binned residual plotfor the exemplary data, where the data points (pairs) are binned bytheir posterior mean probability of the viewer choosing the left image.Differences between the actual proportion of viewers who chose the leftimage in each bin and the proportions predicted by the model arecentered near zero, with no discernible pattern. Further, the 50% and95% intervals exhibit the predicted coverage. The binned residual plotdemonstrates a strong indication that the model fits well with respectto data that comes from the same population as the data to which themodel was fit. Note, however, that when making these predictions, weknew the identities of the viewers of the holdout pairs, we knew thereference images, and we knew the distortion types; this is unlikely tobe true for paired-comparison predictions “in the wild.”

In one or more embodiments, the RMSE can be calculated for modelpredictions of the percentage of pairs in which the viewer _(pre)fersthe left image, for bins with n_(bin)=100 pairs. Referring again to FIG.4A and the exemplary data, the RMSE can be found as the standarddeviation of the differences between the plotted black points and thehorizontal line at zero. For the holdout data, the RMSE for n_(bin)=100is 3.5%, and the errors are approximately normally distributed. Theimplication is that about ⅔ of the time the model's calculatedprediction is found to be within 3.5% of the true percentage. FIG. 4Bdepicts the binned residual plot for the validation set, where the RMSEis 4.9%, and there is a slight pattern of shrinkage in the residuals. Inone or more embodiments, a misclassification error can be calculated,where a posterior mean of P(Y_(i) ^(holdout)=1) can be used to classifyeach pair as either having its left or right image chosen by the viewer.For the exemplary data, the misclassification rate was 22.8% for thepairs in the holdout set. By way of comparison, in another exemplarysample of 400 image pairs labeled by two experts, it was found that theexperts disagreed on 16% of these image pairs.

In one or more embodiments, the server 130 can used the model to makepredictions for new image pairs outside the training population. Thebasic principle is simple: in the absence of knowing an effect of aparticular variable, such as the viewer effect, it is necessary tosample an effect from a distribution of that variable's effects asestimated from the training data. For example, the viewer effectdistribution N(μ^(viewer), σ_(viewer) ²) can be sampled for the viewereffect in the event that the image is being viewed by a viewer who isnot part of the training population. By sampling from the distribution,uncertainty can be propagated for a given variable's effect through themodel and into the predictions.

FIGS. 5, 6, 7A, and 7B, depict illustrative embodiments of a methodoperating in portions of the system described in FIG. 1. In oneembodiment, θ^((g)) denotes a posterior sample g of a parameter θ in theset of posterior samples G, for g=1, . . . , G, and let the variables Land R denote the Left and Right images in the new pair, respectively.Where the viewer, the reference image, or the distortion type for eitherthe L or R image is not known, then the method begins with step 510,where the k QE values can be computed by the server 130 for the leftimage L[i] of the image pair i. In step 520, the k QE values can becomputed by the server 130 for R[i], the right image of pair i. In step530, viewer effects α_(v) ^((g)), for all g=1 to G, can be assigned bythe server 130 from the distribution according to the viewer.

In step 540, the server 130 can assign reference image effects α_(L[i])^(image(g)) and α_(R[i]) ^(image(g)) for images L[i] and R[i] and forall g=1 to G. In one embodiment, the server 130 can sample the referenceimage effects for the L and R images from their prior distributions.Referring to FIG. 6, further details of step 540 are illustrated. Theserver 130 can determine, in step 604, whether the reference image forthe left image L[i] is a known image to the training database. If theleft reference image is known, then the left reference image effectα_(L[i]) ^(image(g)) is set to the known reference image effect value α₁^(image) for g from 1 to G. However, if the left reference image(g)image L[i] is not known, then the left reference image effect α_(L[i])^(image(g)) is sampled from the reference image effect distribution N(0,s_(image) ^(2(g))), for all g=1 to G.

Once the reference image effect values are obtained in either of step604 or step 612, then the server 103 can determine if the rightreference image R[i] is equal to the left reference image L[i], in step616. If the right reference image R[i] is equal to the left referenceimage L[i], then the server 130 can set the right reference image effectα_(R[i]) ^(image(g)) to the same value as the left reference imageeffect α_(L[i]) ^(image(g)), for g=1, . . . , G, in step 624. If,however, the right reference image R[i] is not the same as the leftreference image L[i], then the server 130 can determine if the rightreference image R[i] is a known image from the training database, instep 620. If so, then the server 130 can set the right image effectα_(R[i]) ^(image(g)) to the known reference image effect value α_(r)^(image) for g from 1 to G, in step 628. If the right reference imageR[i] is not known, then the server 130 can sample the right referenceimage effect α_(R[i]) ^(image(g)) from the reference image effectdistribution N(0, S_(image) ^(2(g))), for all g=1 to G.

Referring again to FIG. 5, in step 550, the server 130 can assigndistortion type effects α_(Dist-L[i]) ^(distortion-type(g)) andα_(Dist-R[i]) ^(distortion-type(g)) and k QE effects, denotedβ_((k,Dist-L[i]) ^(distortion-type(g)) and β_((k,Dist-R[i]))^(objective(g)), for k=1 to K, and for g=1 to G, for the left and rightimages, L[i] and R[i]. In one embodiment, the server 130 can sample thedistortion type effects from their prior distributions. Referring now toFIGS. 7A and 7B, further details of step 550 are illustrated. In step704, the server 130 determines if distortion type of the left image L[i]is known. If the distortion type is not known, then the server 130 cansample the left image distortion type effect α_(distL[i])^(distortion-type(g)) from the distortion type effect distribution N(0,σ_(distortion-type) ^(2(g))), for all g=1 to G, in step 716.Subsequently, the server 130 can sample the left image QE effectβ_((k,Dist-L[i])) ^(objective(g)) from the QE effect distributionN(μ_(k) ^(objective(g)), τ_(objective-dist-k) ^(2(g))), for all g=1 to Gand k=1 to K, in step 720. If, however, the server 130 determines thatthe distortion type of the left image L[i] is known (denote it by d),distortion-type(g) then the server 130 can set the left image distortiontype effects α_(distL[i]) ^(distortion-type(g)) to the same value as theknown distortion type effect α_(d) ^(distortion-type(g)), in step 708,and can set the left image QE effect β_((k,Dist-L[i])) ^(objective(g))to the known QE effect, β_((k,d)) ^(objective(g)).

In step 724, the server 130 can determine if the right distortion typeDist-R[i] is equal to the left distortion type Dist-L[i]. If so, then,in step 728, the server 130 can set the right distortion type effectsα_(distR[i]) ^(distortion-type(g)) to the same values as the leftdistortion type effects α_(distL[i]) ^(distortion-type(g)) for all g=1to G and can set the right image QE effects data β_((k,Dist-R[i]))^(objective(g)) to the same value as the left image QE effects dataβ_((k,Dist-L[i])) ^(objective(g)) for all k=1 to K and for g=1 to G. Ifin step 724, the server 130 determines that the right distortion type(Dist-R[i]) is not equal to the left distortion type (Dist-L[i]), instep 754, the server 130 can determine if the distortion type of theright image R[i] is known. If the right image distortion type is notknown, then the server 130 can sample the right distortion type effectdata α_(distR[i]) ^(distortion-type(g)) from the distortion type effectdistribution N(0, σ_(distortion-type) ^(2(g))), for all g=1 to G, instep 766. Subsequently, the server 130 can sample the right image QEeffect β_((k,Dist-R[i])) ^(objective(g)) from t the QE effectsdistribution N(μ_(k) ^(objective(g)), τ_(objective-dist-k) ^(2(g))), forall g=1 to G, and k=1 to K, in step 770. If, however, the server 130determines that the distortion type of the right image R[i] is known(and equal to d), then the server 130 distortion-type(g) can set theright distortion type effect α_(distR[i]) ^(distortion-type(g)) to thesame value as the known distortion type effect α_(d)^(distortion-type(g)), and in step 758, can set the right image QEeffect β_((k,Dist-R[i])) ^(objective(g)) to the known value β_((k, d))^(objective(g)) in step 762, for all g=1, . . . , G.

Referring again to FIG. 5, in step 560, the server 130 can compute anestimate of P^((g})(Y_(i) ^(new)=1), for all g=1 to G. In step 570, theserver 130 can sample a value of Y^(new) _(i) from each of theP^((g})(Y_(i)=1), for all g=1 to G to derive an estimated preferenceoutcome for each g=1, . . . , G. In one or more embodiments, when eitherthe reference images or the distortion types are known to be the same asthose used in the training data, then the posterior distribution isnarrower, but not as narrow as when all variables are known.

FIG. 8 illustrates density estimates for an exemplary posteriordistribution of P(Y_(i) ^(holdout)=1) for a first case where the viewer,distortion types, and reference images are all known (the solid line)and for a second case where none of these variables is known (unequalreference images and equal distortion types are assumed to match theoriginal training data). For example, for an image pair from the holdoutset in which the viewer of both images is known from the training dataas v=74, where the distortion type is known as noise for each image, andwhere the six QE scores are known for each image. It is found that theserver 130 can predict for this image pair, using information fromtraining, the following viewer and image effects {circumflex over (α)}₇₄^(viewer)≈0.01, {circumflex over (α)}₁₀ ^(image)≈−0.05, and {circumflexover (α)}₂ ^(image)≈−0.10, for example.

Factoring in the QE effects, the posterior mean of the probability ofchoosing the left image can be estimated as 0.35, with a standard errorof about 0.06. However, if an image pair was new, and therefore notassociated directly with training data, and if the reference images weredifferent from each other, but the distortion types for both images werethe same (allowing these values to cancel), then the server 130 couldestimate, using the exemplary data, the probability of choosing the leftimage as 0.38 with a standard error of 0.19. In other words, theprobability for the second situation has moved toward 50%, and theconfidence for the probability estimate is lower, due to reduction ofknown effects.

In another embodiment, the accuracy of the predictions made by theserver 130 for new viewers and new reference images can be verified bymaking predictions on a validation set of data. For example, the Stage 2of the exemplary embodiment can be used for accuracy validation.Following the procedure outlined above, in the exemplary embodiment,G=1200 samples were computed from the posterior distributions of theprobability of choosing the left image in each of the validation imagepairs.

Referring again to FIG. 4B, the binned residual plot for predictionsmade on the validation set for the exemplary data and model isillustrated. A bin size of n_(bin)=100 is used (though only 34 bins areshown), to create a visual comparison to the analysis on the holdout setin FIG. 4A. The RMSE of the predicted percentages is 4.9%, and thepattern of residuals indicates that the model is “shrinking” estimatedprobabilities slightly too far toward 50% for pairs in which thepredicted probability is between 20% and 40%. This “shrinking” effectcan be due to variation between viewers or reference images in Stage 2and those in Stage 1, and can cause the estimates of α_(viewer) andα_(image) from the training set to be too large, while possibleintroducing extra randomness into predictions. The misclassificationrate for the validation set was 19.6% for the exemplary data, which issubstantially lower than that of the holdout set. The misclassificationrate with respect to four classes of pairs was found to be 19.1%, 24.2%,15.2%, and 17.8%.

Upon reviewing the aforementioned embodiments, it would be evident to anartisan with ordinary skill in the art that said embodiments can bemodified, reduced, or enhanced without departing from the scope andspirit of the claims described below. In one or more embodiments, theevaluation of the QEs can be performed by devices other than the server130, including in a distributed environment and/or utilizing CPE.

FIG. 9 depicts an exemplary diagrammatic representation of a machine orcontroller circuit in the form of a computer system 900 within which aset of instructions, when executed, may cause the machine to perform anyone or more of the methods discussed above. One or more instances of themachine can operate, for example, as the server 130 as described above.In some embodiments, the machine may be connected (e.g., using anetwork) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server or a client user machine inserver-client user network environment, or as a peer machine in apeer-to-peer (or distributed) network environment.

The machine may comprise a server computer, a client user computer, apersonal computer (PC), a tablet PC, a smart phone, a laptop computer, adesktop computer, a control system, a network router, switch or bridge,or any machine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. It will beunderstood that a communication device of the present disclosureincludes broadly any electronic device that provides voice, video ordata communication. Further, while a single machine is illustrated, theterm “machine” shall also be taken to include any collection of machinesthat individually or jointly execute a set (or multiple sets) ofinstructions to perform any one or more of the methods discussed herein.

The computer system 900 may include a processor 902 (e.g., a centralprocessing unit (CPU), a graphics processing unit (GPU, or both), a mainmemory 904 and a static memory 906, which communicate with each othervia a bus 908. The computer system 900 may further include a videodisplay unit 910 (e.g., a liquid crystal display (LCD), a flat panel, ora solid state display. The computer system 900 may include an inputdevice 912 (e.g., a keyboard), a cursor control device 914 (e.g., amouse), a disk drive unit 916, a signal generation device 918 (e.g., aspeaker or remote control) and a network interface device 920.

The disk drive unit 916 may include a tangible computer-readable storagemedium 922 on which is stored one or more sets of instructions (e.g.,software 924) embodying any one or more of the methods or functionsdescribed herein, including those methods illustrated above. Theinstructions 924 may also reside, completely or at least partially,within the main memory 904, the static memory 906, and/or within theprocessor 902 during execution thereof by the computer system 900. Themain memory 904 and the processor 902 also may constitute tangiblecomputer-readable storage media.

Dedicated hardware implementations including, but not limited to,application specific integrated circuits, programmable logic arrays andother hardware devices can likewise be constructed to implement themethods described herein. Applications that may include the apparatusand systems of various embodiments broadly include a variety ofelectronic and computer systems. Some embodiments implement functions intwo or more specific interconnected hardware modules or devices withrelated control and data signals communicated between and through themodules, or as portions of an application-specific integrated circuit.Thus, the example system is applicable to software, firmware, andhardware implementations.

In accordance with various embodiments of the present disclosure, themethods described herein are intended for operation as software programsrunning on a computer processor. Furthermore, software implementationscan include, but not limited to, distributed processing orcomponent/object distributed processing, parallel processing, or virtualmachine processing can also be constructed to implement the methodsdescribed herein.

While the tangible computer-readable storage medium 922 is shown in anexample embodiment to be a single medium, the term “tangiblecomputer-readable storage medium” should be taken to include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or more sets ofinstructions. The term “tangible computer-readable storage medium” shallalso be taken to include any non-transitory medium that is capable ofstoring or encoding a set of instructions for execution by the machineand that cause the machine to perform any one or more of the methods ofthe present disclosure.

The term “tangible computer-readable storage medium” shall accordinglybe taken to include, but not be limited to: solid-state memories such asa memory card or other package that houses one or more read-only(non-volatile) memories, random access memories, or other re-writable(volatile) memories, a magneto-optical or optical medium such as a diskor tape, or other tangible media which can be used to store information.Accordingly, the disclosure is considered to include any one or more ofa tangible computer-readable storage medium, as listed herein andincluding art-recognized equivalents and successor media, in which thesoftware implementations herein are stored.

Although the present specification describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the disclosure is not limited to such standards andprotocols. Each of the standards for Internet and other packet switchednetwork transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP) representexamples of the state of the art. Such standards are from time-to-timesuperseded by faster or more efficient equivalents having essentiallythe same functions. Wireless standards for device detection (e.g.,RFID), short-range communications (e.g., Bluetooth, WiFi, Zigbee), andlong-range communications (e.g., WiMAX, GSM, CDMA) are contemplated foruse by computer system 800.

The illustrations of embodiments described herein are intended toprovide a general understanding of the structure of various embodiments,and they are not intended to serve as a complete description of all theelements and features of apparatus and systems that might make use ofthe structures described herein. Many other embodiments will be apparentto those of skill in the art upon reviewing the above description. Otherembodiments may be utilized and derived therefrom, such that structuraland logical substitutions and changes may be made without departing fromthe scope of this disclosure. Figures are also merely representationaland may not be drawn to scale. Certain proportions thereof may beexaggerated, while others may be minimized. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense.

Although specific embodiments have been illustrated and describedherein, it should be appreciated that any arrangement calculated toachieve the same purpose may be substituted for the specific embodimentsshown. This disclosure is intended to cover any and all adaptations orvariations of various embodiments. Combinations of the aboveembodiments, and other embodiments not specifically described herein,will be apparent to those of skill in the art upon reviewing the abovedescription.

The Abstract of the Disclosure is provided with the understanding thatit will not be used to interpret or limit the scope or meaning of theclaims. In addition, in the foregoing Detailed Description, it can beseen that various features are grouped together in a single embodimentfor the purpose of streamlining the disclosure. This method ofdisclosure is not to be interpreted as reflecting an intention that theclaimed embodiments require more features than are expressly recited ineach claim. Rather, as the following claims reflect, inventive subjectmatter lies in less than all features of a single disclosed embodiment.Thus the following claims are hereby incorporated into the DetailedDescription, with each claim standing on its own as separately claimedsubject matter.

What is claimed is:
 1. A server comprising: a memory having executableinstructions; and a processor coupled with the memory, wherein theprocessor when executing the executable instructions performs operationscomprising: receiving a request over a network to predict an imagepreference for an image pair comprising a first image and a secondimage; identifying a first distortion type associated with the firstimage and a second distortion type associated with the second image;sampling a variable effect distribution of viewing preference data todetermine a first set of effects comprising a plurality of firstreference image effects associated with the first image and a pluralityof first distortion type effects associated with the first distortiontype and to determine a second set of effects comprising a plurality ofsecond reference image effects associated with the second image and aplurality of second distortion type effects associated with the seconddistortion type; calculating a preference estimate from a logisticregression model of the viewing preference data according to the firstset of effects and the second set of effects, wherein the preferenceestimate comprises a probability that the first image is preferred overthe second image; and sending the preference estimate over the network.2. The server of claim 1, wherein the processor further performsoperations comprising: determining whether the first distortion typematches a distortion type of the variable effect distribution; andsetting the plurality of first distortion effects to a first distortioneffect value according to the first distortion type that is matched. 3.The server of claim 2, wherein the processor further performs operationscomprising setting a plurality of quality estimator effects to a firstquality estimator value according to the first distortion type that ismatched, wherein the calculating of the preference estimate is furtheraccording to the plurality of quality estimator effects.
 4. The serverof claim 1, wherein the processor further performs operationscomprising: determining whether the first image matches a referenceimage of the variable effect distribution; and setting the plurality offirst reference image effects to a first reference image effect valueaccording to the first image that is matched.
 5. The server of claim 4,wherein the processor further performs operations comprising:determining whether the second image matches the first image; andsetting the plurality of second reference image effects to the firstreference image effect value according to the second image matching thefirst reference image.
 6. The server of claim 1, wherein the processorfurther performs operations comprising identifying a first viewer of thefirst image and a second viewer of the second image, wherein the firstset of effects further comprises a plurality of first viewer effectsassociated with the first viewer and wherein the second set of effectsfurther comprises a plurality of second viewer effects associated withthe second viewer, and wherein the calculating of the preferenceestimate is further according to the plurality of first viewer effectsand the plurality of second viewer effects.
 7. The server of claim 6,wherein the processor further performs operations comprising:determining whether the first viewer matches a viewer of the variableeffect distribution; and setting the plurality of first viewer effectsto a first viewer effect value according to the first viewer that ismatched.
 8. The server of claim 6, wherein the processor furtherperforms operations comprising: determining whether the first viewer andthe second viewer are a common entity; and setting the first viewereffect and the second viewer effect to a common value.
 9. The server ofclaim 1, wherein the processor further performs operations comprisingselecting one of the first distortion type or the second distortion typeaccording to the preference estimate.
 10. The server of claim 1, whereinthe first distortion type comprises one of a Gaussian blur, an imagecompression, an additive Gaussian noise, or a combination thereof. 11.The server of claim 1, wherein the variable effect distribution ofviewing preference data is generated from logging image pair preferenceobservations from a plurality of viewers of a plurality of image pairs,wherein the plurality of image pairs are associated with a plurality ofreference images that are modified via a plurality of distortion types.12. The server of claim 1, wherein the processor further performsoperations comprising: calculating a confidence value according to thefirst set of effects and the second set of effects, wherein theconfidence value further defines the preference estimate; and sendingthe confidence value over the network.
 13. The server of claim 1,wherein the logistic regression model of the viewing preference datacomprises a Bayesian regression model.
 14. The server of claim 1,wherein the variable effect distribution of viewing preference datacomprises a plurality of normal distributions.
 15. A method comprising:identifying, by a processor comprising a device, a first distortion typeassociated with a first image of an image pair and a second distortiontype associated with a second image of the image pair; sampling, by theprocessor, a variable effect distribution of viewing preference data todetermine a first set of effects comprising a plurality of firstdistortion type effects associated with the first distortion type and todetermine a second set of effects comprising a plurality of seconddistortion type effects associated with the second distortion type;calculating, by the processor, a preference estimate from a logisticregression model of the viewing preference data according to the firstset of effects and the second set of effects, wherein the preferenceestimate comprises a probability that the first image is preferred overthe second image; and selecting, by the processor, a distortion type forprocessing media content from a plurality of distortion types accordingto the preference estimate.
 16. The method of claim 15, wherein thefirst set of effects further comprises a plurality of first referenceimage effects associated with the first image, wherein the second set ofeffects further comprises a plurality of second reference image effectsassociated with the second image.
 17. The method of claim 15, whereinthe first distortion type comprises one of a Gaussian blur, an imagecompression, an additive Gaussian noise, or a combination thereof. 18.The method of claim 15, further comprising calculating, by theprocessor, a confidence value according to the first set of effects andthe second set of effects, wherein the confidence value further definesthe preference estimate, wherein the selecting the distortion type isfurther according to the confidence value.
 19. A computer-readablestorage device comprising executable instructions, which when executedby a processor cause the processor to perform operations comprising:sampling a variable effect distribution of viewing preference data todetermine a first set of effects comprising a plurality of firstdistortion type effects associated with a first distortion type of afirst image and to determine a second set of effects comprising aplurality of second distortion type effects associated with the seconddistortion type of a second image; calculating a preference estimatefrom a logistic regression model of the viewing preference dataaccording to the first set of effects and the second set of effects,wherein the preference estimate comprises a probability that the firstimage is preferred over the second image; and selecting one of the firstdistortion type or the second distortion type according to thepreference estimate.
 20. The storage device of claim 19, furthercomprising executable instructions that cause the processor to performoperations comprising calculating a confidence value according to thefirst set of effects and the second set of effects, wherein theconfidence value further defines the preference estimate, wherein theselecting is further according to the confidence value.